A General Theory of Canonical Forms

If G is a compact Lie group and M a Riemannian G-manifold with principal orbits of codimension k then a section or canonical form for M is a closed, smooth k-dimensional submanifold of M which meets all orbits of M orthogonally. We discuss some of the remarkable properties of G-manifolds that admit sections, develop methods for constructing sections, and consider several applications. O. Introduction. Let G be a compact Lie group acting isometrically on a Riemannian manifold M. Then the image S of a small ball in the normal plane v{Gx)x under the exponential map is a smooth, local Gx-slice, which in general cannot be extended to a global slice for M. A section E for M is defined to be a closed, smooth submanifold of M which meets every orbit of M orthogonally. A good example to keep in mind is perhaps the most important of all canonical form theorems; namely for M we take the Euclidean space of symmetric k x k matrices with inner product (A, B) = tr(AB), and for G the orthogonal group O{k) acting on M by conjugation. Then the space E of diagonal matrices is a section. Moreover the symmetric group Sk acts on E by permuting the diagonal entries and the orbit spaces M/G and E/Sk are isomorphic as stratified sets. Quite generally it is good intuition to think of a section E as representing a "canonical form" for elements of M; hence our title. Riemannian G-manifolds which admit sections are definitely the exception rather than the rule and they have many remarkable properties. The existence of sections for M has important consequences for the invariant function theory, submanifold geometry, and G-invariant variational problems associated to M. While we do not know of earlier papers treating sections in generality, we have found several which treat important special cases. In particular when we showed G. Schwarz an early version of our results he pointed out to us a preprint of an important paper [Da2] by J. Dadok in which a detailed study is made (including a complete classification theorem) of orthogonal representations of compact connected Lie groups which admits sections (Dadok calls these polar representations). Later still we discovered two very interesting and much earlier papers by L. Conlon [Col, Co2] in which he considers Riemannian G-manifolds which admit fiat, totally geodesic sections. This includes the case of polar representations, and Conlon came close to conjecturing Dadok's classification result. We will discuss in more detail later the results in these papers and how they relate to our own. We would like to thank Dadok for a number of helpful comments. It is clear not only that he Received by the editors March 4, 1986. 1980 Mathematics Subject Classification (1985 Revision). Primary 57S15. The first author was supported in part by NSF Grant No. MCS-8102696 and MSRI. The second author was supported in part by NSF Grant No. DMS-8301928. 771 @1987 American Mathematical Society 0002-9947/87 $1.00 + $.25 per page 772 R. S. PALAIS AND CHUU-LIAN TERNG had also discovered many of the facts reported here, but in some cases he probably knew them before we did. One important application of sections is to invariant theory. We describe a wellknown class of examples: if a compact Lie group G acts on its Lie algebra g (Killing form as the inner product) by the adjoint action, then a maximal abelian sub algebra T is a section, the Weyl group W of G acts on T, and giG ~ T IW. Moreover the restriction map from the ring of G-invariant polynomials on g to the ring of W-invariant polynomials on T is an isomorphism; this is the Chevalley restriction theorem. These properties of the adjoint action can be generalized to arbitrary G-manifolds which admit sections. Namely if E is a section of a Riemannian Gmanifold M, then there exists a finite group W acting on E such that for each 0" in E, En GO" = WO" (so in particular WO" ~ GO" is a bijection EIW ~ MIG); and the Coo version of the Chevalley restriction theorem holds. This reduces the theory of G-invariant functions on M to the simpler invariant theory of T under a finite group. A second application is to the Riemannian geometry of submanifolds. The principal horizontal distribution ){ is a distribution defined on the set MO of G-regular points by ){(x) = v(Gx)x. Then it is easily seen that M admits sections if and only if){ is integrable and expx(v(Gx)x) is a closed, properly embedded submanifold of M (which is automatically totally geodesic). If v E v(Gx)x then v(gx) = dgx(v) defines a G-equivariant normal field on the principal orbit Gx, and we say v is 7r-parallel. Since Gx is a submanifold of M, there is an induced normal connection from the Riemannian connection of M, which defines another parallelism for v(Gx). In general these two parallelisms are different, and in fact they are the same if and only if ){ is integrable. In this case a principal orbit N = Gx as a submanifold of M has the following properties: (1) v( N) is flat with trivial holonomy, (2) expy(v(N)y) is a totally geodesic submanifold of M for all y in N, (3) the principal curvatures of N along any parallel normal field are constant. We note that the orbit foliation of M is determined by a single principal orbit, and is the same as the parallel foliation of N in M, i.e. {Nv Iv E v( N)x}, where Nv = {y + v(y)ly EM}. A submanifold N of a space form Rn satisfying (i)-(iii) is called isoparametric [T]. So it follows that if G acting on Rn admits sections then the principal orbits are isoparametric. Conversely we show that if N is isoparametric in Rn and is an orbit of a subgroup G of O(n) then N must be a principal G-orbit and the G-action on Rn admits sections. Then by Dadok's classification theorem of polar representation we conclude that every homogeneous isoparametric submanifold of Rn or sn is a principal orbit of the isotropy representation of some symmetric space Gd K. There are infinitely many isoparametric submanifolds of Euclidean spaces of co dimension two, which do not arise as an orbit of some polar representation [FKM, OT]. However there always exists a Weyl group for such submanifolds, and the parallel foliation gives an orbitlike foliation. Therefore the theory of isoparametric submanifolds can be generalized to arbitrary Riemannian manifolds (using (1)-(3) as definition), which can be thought of as a purely geometric analogue of the theory of Riemannian G-manifolds with sections. The third application we have in mind is to the calculus of variations. We first recall the simple example of finding a harmonic function u on Rn. We must in A GENERAL THEORY OF CANONICAL FORMS 773 principle solve a partial differential equation ~u = 0 in n-independent variables. However if we know that u is invariant under the group O(n) of rotations, then we can write u(x) = f(llxll) and reduce the problem to the easily solved ordinary differential equation 8j8r(rn 18f j8r) = 0 on the half line A+. This is a classic example of a general and powerful method, called variously "reduction of variables" or the "cohomogeneity method" , for attacking a broad class of problems in geometry and analysis (cf. [HHS, Hsl, Hs2, HL, PT]). In the general setting, we have as above a G-manifold Mn and would like to study some class (5 of G-invariant objects associated to M. Frequently one can set up a natural bijection between (5 and some set 6 of related objects attached to the orbit space M, so that if M has cohomogeneity k (i.e. dim(M) = k) we have effectively reduced a problem with n independent variables to a generally easier problem with only k independent variables. A serious difficulty in applying this method comes from the existence of the set Ms of singular (i.e. lower dimensional) orbits. In general M is not a smooth manifold but only a stratified set. The principal stratum M Ms (the set of principal orbits) is an open, dense, smooth k-dimensional manifold. Ms is the (finite) union of the other orbit types of M, each of which is by itself a smooth manifold of dimension less than k, but in general M has bad singularities along Ms , making it hard to study global analytical problems on M. The study of 6 usually leads to solving some partial differential equation on M Ms together with complicated "boundary behavior" as we approach Ms. To circumvent the difficulties associated to the latter one can try to "resolve" the singularities along Ms , and an excellent way to do this is to choose (if one exists) a section E for M as above. Then the analysis of 6 leads to solving a partial differential equation on the smooth k-manifold E (rather than on the singular k-manifold M) and the complicated boundary behavior along Ms is replaced by the generally more tractable problem of W -invariance. (For example in our example of harmonic functions on M = An with G = O(n), where M = Ill+ = [0,00) and Ms = {O}, we can take for E any line {relr E Fil} with e in sn-l and W = 12 (generated by re ----; -re), so that instead of solving 8j8r(rn 1(8f j8r)) = 0 on R+ with certain boundary behavior at 0, we solve it on III but accept only even solutions). It is not hard to see that our definitions and theorems concerning Riemannian G-manifolds with sections generalize easily if we drop the assumption that the Lie group G is compact and replace it with the weaker assumption that G acts properly on M (which is equivalent to the condition that there exists a G-invariant Riemannian metric on M with G being a closed subgroup of Iso(M)). Because many variational problems in geometry and physics are invariant under an infinite dimensional Lie group of "gauge transformations", another very interesting direction of generalization, about which little is yet known, is to develop an analogous canonical form theory for infinite dimension manifolds with infinite dimensional Lie group actions. In the case of the group of diffeomorphisms acting on the space of Riemannian metrics and the group 9 of gauge transformations acting on the space .A of connections of a principal bundle it is known that the actions are proper and that they admit local slices, so the possibility of sections existing, at least in special cases, seems quite reasonable. Moreover in the latter case doing a path integral over a section would clearly be easier than doing one over the moduli space .A j g. This paper is organized as follows: we set the terminology and review basic 774 R. S. PALAIS AND CHUU-LIAN TERNG properties of G-manifolds and Riemannian G-manifolds in §1 and §2, and in §3 we develop some elementary properties of sections; we discuss the generalized Weyl group and the Coo Chevalley restriction theorem for a Riemannian G-manifold which admits sections in §4; and in §5 we prove that if M is a G-manifold and the principal isotropy subgroup H is open in its normalizer N (H) then the fixed point set E of H is a section with respect to any G-invariant metric on M, i.e. the section depends only on the pair (G, H). Finally in §6 we discuss the submanifold geometry of the orbits of Riemannian G-manifolds M which admit sections. 1. G-manifolds. In this section we establish our notation and review the basic theory of smooth transformation groups. Most details and proofs are omitted and may be found in [B, D, and 82]. G will denote a compact Lie group and M a connected, smooth (i.e. COO) Gmanifold. For x in M we denote its orbit by Gx and its isotropy group by G x . We denote the orbit space M/G with the identification space topology by M and II: M ----> M is the orbit map. The conjugacy class of a closed subgroup H of G will be denoted by (H) and is called a G-orbit type; the orbit Gx is said to be of type (H) if (Gx) = (H), M(H) ~ M denotes the union of all orbits of type (H), and M(H) ~ M its image in M (the set of all orbits of type (H)). The fixed point set of H, i.e. the set of those x in M with H ~ Gx , will be denoted as usual by M H, and M H will denote the set of x in M where G x is equal to H (so MH = (M(H))H = M(H) nMH). From the fact that Ggx = gGxg1 it follows that gMH = MgHg-l. On the other hand if N(H) denotes the normalizer of H in G then gHg1 depends only on the coset gN(H) of gin G/N(H). It follows that we have a well-defined map p: M(H) ----> G/N(H) with p-l(gN(H)) = MgHg-l. In fact it is not hard to see that each M(H) is a smooth regularly embedded (but usually not closed) submanifold of M, and that p is a smooth fiber bundle with fiber MH associated to the principal fibration G ----> G/N(H), with structure group N(H)/ H. In case M has a single orbit type there is a canonical differentiable structure for M making II: M ----> M a submersion (and in fact a fiber bundle). But as just remarked, in general each M(H) is a smooth submanifold of M and hence a smooth G-manifold in its own right, so each M(H) has a canonical differentiable structure. In fact these decompositions of M and Minto submanifolds are "nice" (technically M and M are both stratified sets and II: M -+ M is a stratified submersion). This fact has played an important role in the recent history of the subject. We refer to [D] for details. Among all orbit types (H) with M(H) f:. 0 there is a unique one (U) such that G/U has maximum dimension and (for that dimension) a maximum number of components. The orbit type (U) is called the principal orbit type of M, any representative U is called a principal isotropy group, and M(u) is called the principal stratum of M. To avoid having to name (U) we will also write MO and MO for M(u) and M(u) respectively. The nonprincipal orbits of M are called singular orbits and their union Ms (the complement of MO) is called the singular set of M. Thus Ms and Ms are closed and nowhere dense in M and M respectively. Points of MO are called regular points and points of Ms singular points. By choosing any Riemannian structure for M and averaging it with respect to the Haar measure of G we can always find an invariant Riemannian metric for M, A GENERAL THEORY OF CANONICAL FORMS 775 i.e. one for which G acts by isometries. Such a metric is also called compatible with the action of G, and M with such a metric is called a Riemannian G-manifold. The differential of the action of Gx defines a linear representation of Gx on T Mx called the isotropy representation at x. Since the tangent space T( Gx)x to the orbit of x is clearly an invariant linear subspace, we can find a complementary invariant subspace v(Gx)x (e.g. the orthogonal complement to T(Gx)x with respect to a compatible metric), and the restriction of the isotropy representation of Gx to v(Gx)x is called the slice representation at x. The image of a small ball in v(Gx)x under the exponential map (with respect to a compatible metric) is a smooth Gx invariant disk 6 in M called a slice at x. It meets all nearby orbits transversally and has the important property that for y in 6 the isotropy group Gy is included in Gx . It follows easily that x is a regular point if and only if the slice representation is trivial, or equivalently if and only if 6 is pointwise fixed under Gx . 2. Riemannian submersions and Riemannian G-manifolds. If II: E -t B is a submersion of smooth manifolds then V = ker( dII) is a smooth subbundle of T E called the tangent bundle along the fiber (or the vertical sub bundle). In case E and B are Riemannian we define the horizontal subbundle }I of T E to be the orthogonal complement V..L of the vertical bundle, and II is called a Riemannian submersion if dII maps }Ix isometrically onto T Bn(x) for all x in E. The theory of Riemannian submersions, first systematically studied by O'Neill [0], plays an important role in the study of transformation groups. In this section we will discuss some basic geometric properties of Riemannian submersions and Riemannian Gmanifolds. A vector field ~ on E is called vertical (resp. horizonta0 if ~(x) is in V(x) (resp. }I (x)) for all x in E, and ~ is called projectable if there exists a vector field 17 on B such that dII(~) = 17. We call ~ basic if it is both horizontal and projectable. Note that if F = II-l(y) is a fiber of II then }l1F is just the normal bundle v(F) to F in E. There is a canonical global parallelism in each such normal bundle v(F): a section v of v(F) is called II-parallel if dII(v(x)) is a fixed vector v E TBy independent of x in F. Clearly v -t v is a bijective correspondence between IIparallel fields and T By. There is another standard parallelism on v( F) obtained from the Riemannian structure of E. Let V denote the Levi-Civita connection of E, then the induced normal connection V on v(F) is defined as follows: V x~ = the orthogonal projection on V x~ onto v(F), where X E COO(TF) and ~ E COO(v(F)). A normal vector field ~ on F is called parallel if V ~ = O. It is important to note that in general the II-parallelism in v(F) has no relation to the parallel translation defined by the Riemannian connection in v(F). (The latter is in general not flat, while the former is always both flat and without holonomy.) Nevertheless we shall see that if }I is integrable then these two parallelisms do coincide. To prove this we need some basic results in the theory of Riemannian submersions. 2.1. THEOREM (0' NEILL [0]). Let II: E -t B be a Riemannian submersion, and }I its horizontal distribution. (i) If X is a vertical field and Y is a basic field on E then [X, Y] is vertical. (ii) If (J is a geodesic inE and (J'(O) is horizontal then (J'(t) is horizontal for all t and II 0 (J is a geodesic in B. 776 R. S. PALAIS AND CHUU-LIAN TERNG (iii) If}.( is integrable then the leaves are totally geodesic. (iv) If}.( is integrable and S is a leaf of ).( then IllS is a local isometry. 2.2. DEFINITION. A Riemannian submersion II: E -+ B is called integrable if the horizontal distribution ).( is integrable. 2.3. THEOREM. Let II: E -+ B be a Riemannian submersion. Then II is integrable if and only if every II-parallel vector field on II-l(b) is a parallel normal field in the Riemannian sense (i. e. it is parallel with respect to the induced normal connection of II1 (b)). PROOF. Let eA be a local orthonormal frame field on E such that el, ... , en are basic vector fields and en + 1, ... ,en+m are vertical vector fields. We will use the index conventions 1 :S i, j :S n, n + 1 :S 0:, f3 :S n + m, 1 :S A, B :S n + m, and we will write 'Vi for 'Vei , ••• , etc. Let WAB be the Levi-Civita connection on M, i.e. 'VeA = L:wAB ® eB, and suppose II is integrable. Then by 2.1(iii) each leaf S of the horizontal distribution ).( is totally geodesic and eilS is a local frame field on S. Thus the second fundamental form of S is zero, i.e. Wia (ej) = 0 for all i and j, or equivalently 'V jea is vertical. But ealF forms a tangent frame field for the fiber F of II, and ei!F is a normal vector field of F. Since [ej, ea] = 'Vjea 'Vaej is vertical, we have 'Vaej is vertical, i.e. ejlF is parallel in the normal connection of v(F). Conversely suppose ei IF is parallel for every fiber F of II, i.e. 'Vaej is vertical. Since [ei,ea] is vertical, 'Vjea is vertical, i.e. wai(ej) = 0 for all i and j. Now we note that lei, ej] = 'Viej 'Vjei = ~)WjA(ei) wiA(ej))eA. Hence lei, ej] is horizontal, so ).( is integrable. • Henceforth M will denote a connected, complete smooth Riemannian G-manifold. As noted in the preceding section for each orbit type (K) the restricted orbit map II(K): M(K) -+ M(K) is a submersion. We note that there is a canonical choice of Riemannian structure for M(K) making II(K) a Riemannian submersion (so that II: M -+ M is a stratified Riemannian submersion). To see this we can without loss of generality assume M = M(K). If x E M then we must of course define the inner product in TMrr(x) by requiring that dII: v(Gx)x -+ TMrr(x) is an isometry. Since dg maps v(Gx)x isometrically onto v(Gx)gX this is well defined and is easily seen to give a smooth metric on M. Thus in particular we have a Riemannian submersion on the principal stratum II: MO -+ MO. 2.4. DEFINITION. The principal horizontal distribution of a Riemannian Gmanifold M is the horizontal distribution of the Riemannian submersion on the principal stratum II: MO -+ MO. If x is a regular point of M then the orbit Gx is a fiber of II and hence we have as above a well-defined global parallelism in v(Gx), the II-parallelism. In this case the II-parallelism has a simple group theoretic interpretation. Since x is regular, the slice representation of Gx on v(Gx)x is trivial, which implies that dg: v( Gx)x -+ v(Gx)gX is well defined (i.e. does not depend on the choice of element in the coset gGx). Thus any element Vx E v(Gx)x gives rise to a well-defined Ginvariant section v of v( Gx). Moreover since II 0 9 = II, dII 0 dg = dII and hence v is II-parallel, i.e. the II-parallelism is J'ust given by group translation. (A word of A GENERAL THEORY OF CANONICAL FORMS 777 caution: for a non principal stratum M(H) we again have a Riemannian submersion IT(H): M(H) ---M(H) and hence a IT(H)-parallelism on the normal bundle lI(H) (Gx) of an orbit Gx of type (H). But note that V(H)(GX) denotes the normal bundle of Gx in M(H), a subbundle of v(Gx), its normal bundle in M.) It will be convenient to introduce for each regular point x the set T (x), defined as the image of v(Gx)x under the exponential map of M and also TO(x) = T(x) nMo for the set of regular points of T (x). 2.5. PROPOSITION. For each regular point x of M: (i) gT(x) = T(gx) and gTO(x) = TO(gx) for all 9 E G, (ii) for Vo E v(Gx)x the geodesic O'(t) = exp(tvo) is orthogonal to each orbit it meets, (iii) T(x) meets every orbit of M. PROOF. Statement (i) is obvious, and (ii) follows from 2.1(ii) and the fact that IT: MO ____ £10 is a Riemannian submersion. Finally given any y in M choose 9 E G so that gy minimizes the distance from x to Gy and define O'(t) = exp(tvo), a minimizing geodesic from x = 0'(0) to gy = 0'(1). Since G acts isometrically, 0' is even a minimizing geodesic from Gx to Gy, and hence Vo = 0"(0) and 0"(1) are orthogonal to Gx and Gy respectively. In particular Vo is in 1I( Gx)x so the arbitrary orbit Gy meets T(x) = exp(lI(Gx)x) at exp(vo) = gy. • Our choice of the notation T (x) is based on the fact that when M is G itself with the adjoint action, then T (x) is just the maximal torus through the regular point x. Thus (iii) is a generalization of the fact that every element of It compact connected Lie group is conjugate to an element on a fixed maximal torus. 3. Sections and their elementary properties. Henceforth M will denote a connected, complete, Riemannian G-manifold and we assume all the previous notational conventions. In particular we identify the Lie algebra 9 of G with the Killing fields on M generating the action of G. 3.1. DEFINITION. A connected, closed, regularly embedded smooth submanifold ~ of M is called a section for M if it meets all orbits orthogonally. The conditions on ~ are, more precisely, that G~ = M and that for each x in ~, T~x is included in v(Gx)x = T(Gx);. But since T(Gx)x is just the set of ~(x) where ~ E g, this second condition has the more explicit form (*) For each x in ~ and ~ in g, ~(x) is orthogonal to T~x. It is trivial that if ~ is a section for M then so is g~ for each 9 in G. Since G~ = M, it follows that if one section ~ exists then in fact there is a section through each point of M, and we shall say that M admits sections. If ~ is a section for M then the set ~o = ~ n MO of regular points of ~ is an integral manifold of the principal horizontal distribution ).I of the G-action. It is known (see [D, Theorem 1.7]) that £10 is always connected, so from 2.3 it follows